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ABSTRACT 

This empirical study aims to demonstrate how Dispositional Learning Analytics can contribute in the investigation of the 
effectiveness of didactical scenarios in authentic settings, where previous research has mostly been laboratory based. 
Using a showcase based on learning processes of 1080 students in a blended introductory quantitative course, we analyse 
the use of worked examples by students. Our method is to combine demographic and trace data from technology 
enhanced systems with self-reports of several contemporary social-cognitive theories. We find that the same maladaptive 
learning orientations that play a role in worked examples learning theories as to explain the effectiveness of worked 
examples do predict the use of worked examples: this time in the role of individual learning dispositions. 


KEYWORDS 

Blended learning; dispositional learning analytics; e-tutorials; learning feedback; learning dispositions; worked examples 


1. INTRODUCTION 

Although there exists little misunderstanding on what learning Analytics (LA) is supposed to study, LA 
practices seem to be at a large distance. ‘ Learning analytics uses dynamic information about learners and 
learning environments, assessing, eliciting and analysing it, for real-time modelling, prediction and 
optimisation of learning processes, learning environments and educational decision-making ’ (Ifenthaler, 
2015) is a broadly accepted description of all facets that are included in LA. In the same source, this 
definition is elaborated by enumerating all ten essential components of a holistic LA framework: information 
about the learners’ individual characteristics, information from the social web, physical data, information 
from learners’ activities in the online learning environment, curriculum information, the learning analytics 
engine, the reporting engine, and the personalisation and adaption engine (Ifenthaler, 2015). Yet, current LA 
practices typically incorporate only a few of these ten components, with a strong focus on building predictive 
models based on demographics, grades, and trace data. 

To emphasize the importance of other data than trace or system data, Buckingham Shum and Crick 
(2012) proposed a Dispositional Learning Analytics (DLA) infrastructure that combines learning data 
(generated in learning activities through technology enhanced systems) with learner data (student 
dispositions, values, and attitudes measured through self-report surveys). Learning dispositions represent 
individual difference characteristics that impact all learning processes and include affective, behavioural and 
cognitive facets (Rienties, Cross, S., Zdrahal, 2017). Student’s preferred learning approaches are examples of 
such dispositions of both cognitive and behavioural type; in research on their role in learning, they are often 
simply labelled as ‘self-report data’, and in the above description of the holistic LA framework, indicated as 
the learners’ individual characteristics component (Ifenthaler, 2015). Different from LA research, 
stakeholders of DLA applications are typically restricted to students and teacher/tutors, as these applications 
can be positioned at both the meso- and micro-level (Ifenthaler, 2015), rather than the mega- or macro-level. 

Our current study aims to provide a show case of the educational benefits of following this more holistic 
approach in the application of LA. We do so in the context of an instructional design issue: the worked 
examples principle (Renkl, 2014). The merits of using worked examples in the initial acquisition of cognitive 
skills are well documented. The evidence is without exception based on laboratory-based experimental 
studies, in which the effectiveness of different instructional designs is compared (Renkl, 2014). In this issue, 
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the potential contribution of LA-based investigations in authentic contexts is that we can observe students’ 
revealed preferences for learning with worked examples, rather than assigning them to a worked examples 
condition, and link the preferences to other observations: information about the learners’ individual 
characteristics, information from the social web, and information from learners’ activities in the online 
learning environment (the information components of Ifenthaler’s holistic framework). By doing so, we aim 
to derive a characterization of students who actively apply worked examples, and those not doing so and link 
these characterizations to the outcomes of existing laboratory-based research. This study is a follow-up of 
previous DLA research by the authors on the use of worked examples (Tempelaar, Rienties, & Nguyen, 
2017a). The focus of that previous research was on the timing of the use of worked examples: in the initial 
skills acquisition, or at a later stage in the learning process. The focus of the current research is on individual 
differences in the intensity of using worked examples. 


2. WORKED EXAMPLES 

The micro level pedagogical benefits that come with LA applications refer to the provision of ‘personalized 
and adaptive scaffolds’ supporting the learner in reaching the learning outcomes (Ifenthaler, 2015). Worked 
examples represent one of the scaffold formats in computer-enhanced environments (Duffy & Azevedo, 
2015), formats that amongst others differ in the amount of guidance or assistance provided to students. 
Pedagogics has identified four main instructional approaches for assisting learners in problem-solving 
(McLaren, Van Gog, Ganoe, Karabinos, & Yaron, 2016), with varying degrees of learner support. First, the 
problem-solving approach is positioned in the low guidance end of the continuum, offering little or no 
feedback to learners. Second, tutored problem-solving provides learners with feedback and hints to solve the 
problem or construct the schema when a learner is stuck. This approach intervenes in the learning process 
only when help is needed; hence, it ensures learners will actively attempt to solve the problems. Third, 
erroneous examples present learners with flawed examples and instinct them to find, explain, and fix errors. 
Finally, at the high end of learner support McLaren et al. (2016) position the use of worked examples. The 
use of worked solutions in multi-media based learning environments stimulates gaining deep understanding 
(Renkl, 2014). When compared to the use of erroneous examples, tutored problem-solving, and 
problem-solving in computer-based environments, the use of worked examples may be more efficient as it 
reaches similar learning outcomes in less time and with less learning efforts (McLaren et al., 2016). The 
mechanism responsible for this outcome is disclosed in Renkl (2014, p. 400): ‘ examples relieve learners of 
problem solving that - in initial cognitive skill acquisition, when learners still lack understanding - is 
typically slow, error-prone, and driven by superficial strategies. When beginning learners solve problems, 
the corresponding demands may burden working memory capacities or even overload them, which 
strengthens learners’ surface orientation. ... When learning from examples, learners have enough working 
memory capacity for self-explaining and comparing examples by which abstract principles can be 
considered, and those principles are then related to concrete exemplars. In this way, learners gain an 
understanding of how to apply principles in problem solving and how to relate problem cases to underlying 
principles'. 

Studies into the efficiency of worked examples are typically nested in laboratory settings, with students 
assigned to one of the several experimental conditions, each representing one unique pedagogical feedback 
scenario. In authentic settings, students mix and match diverse pedagogical feedback scenarios, and do so in 
different orders. For example, some students will avoid using worked examples; other students use just a 
single worked example at the very start of skills acquisition, whereas others apply worked examples exactly 
in the way the theory of example-based learning would indicate: self-explaining and comparing multiple 
examples as to ‘represent principles in the form of abstract schema that are interrelated to multiple example 
cases so that learners know (1) how the abstract principles can be applied and (2) how concrete problem 
cases can be interpreted in terms of underlying principles .’ (Renkl, 2014, p. 400). Beyond detecting 
individual differences in preferences for pedagogical scenarios, a next step is to explain these based on 
differences in learning dispositions. For example, studies in gender differences in learning mathematics 
suggest that female students would profit more from having worked examples available at the very start of 
learning new mathematical concepts (Boltjens, 2004). Do such outcomes of experimental studies transfer to 
differences in revealed preferences in an authentic learning setting? And what other learning dispositions do 
make a difference in the use of worked examples beyond the potential role of gender? 
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3. METHODS 


3.1 Context of the Empirical Study 

This study takes place in a large-scale introductory mathematics and statistics course for first-year 
undergraduate students in a business and economics programme in the Netherlands. The educational system 
is best described as ‘blended’ or ‘hybrid’. The main component is face-to-face: Problem-Based Learning 
(PBL), in small groups (14 students), coached by a content expert tutor (see Williams et al., 2016 for further 
information on PBL and the course design). Participation in tutorial groups is required. Optional is the online 
component of the blend: the use of the two e-tutorials SOWISO (https://sowiso.nL) and MyStatLab (MSL) 
(Tempelaar, Rienties, & Nguyen, 2015, 2017b). This design is based on the philosophy of student-centred 
education, placing the responsibility for making educational choices primarily on the student. Since most of 
the learning takes place during self-study outside class through the e-tutorials or other learning materials, 
class time is used to discuss solving advanced problems. Thus, the instructional format is best characterized 
as a flipped-classroom design (Williams et al., 2016). Using and achieving good scores in the e-tutorial 
practice modes is incentivized by providing bonus points for good performance in quizzes that are taken 
every two weeks and consist of items that are drawn from the same item pools applied in the practicing 
mode. This approach was chosen in order to encourage students with limited prior knowledge to make 
intensive use of the e-tutorials. 

The subject of this study is the hill 2016/2017 cohort of students (1093 students). A large diversity in the 
student population was present: only 19% were educated in the Dutch high school system. In terms of 
nationality, the largest group, 44% of the students, was from Germany, followed by 23% Dutch and 19% 
Belgian students. In total, 50 nationalities were present. A large share of students was of European 
nationality, with only 3.9% of students from outside Europe. High school systems in Europe differ strongly, 
most particularly in the teaching of mathematics and statistics. For example, the Dutch high school system 
has a strong focus on the topic of statistics, but is mostly missing in high school programs of other countries. 
Therefore, it is crucial that this present introductory module is flexible and allows for individual learning 
paths (Williams et al., 2016). In this course, students spend on average 24 hours in SOWISO and 32 hours in 
MSL, which is 30% to 40% of the available time of 80 hours for learning on both topics. 

3.2 Instruments and Procedure 


The empirical analyses described in this contribution focus on the use of the MSL e-tutorial for learning 
statistics. Although Pearson MyLabs can be used as a learning environment in the broad sense of the word (it 
contains, among others, a digital version of the textbook), they represent primarily an environment for 
test-directed learning and practicing. Each step in the learning process is initiated by a question, and students 
are encouraged to (try to) answer each question. If a student does not master a question, she/he can either ask 
for help to solve the problem step-by-step (Help Me Solve This), or ask for a worked example (View an 
Example), as demonstrated in Figure 1 (left panel), in any lesson. 

Section 2.3 Exercise 7 :S Question Help <1 O view an Example X 
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Figure 1. MSL exercise window, left panel, and worked example window, right panel 
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Students can call for multiple examples that differ in the context of the application of the same statistical 
principle, as indicated by the theory of example-based learning (Figure 1, right panel). When after studying 
these examples the student feels ready to make an own attempt, a new version of the problem loads 
(parameter based) to allow the student to demonstrate his/her newly acquired mastery. 

Our study combines trace data of the MSL e-tutorial with self-report survey data measuring learning 
dispositions. Trace data is both of product and process type (Azevedo, Flarley, Trevors, Duffy, 
Feyzi-Behnagh, & Bouchet, 2013). MSL reporting options of trace data are very broad, requiring making 
selections from the data. First, all dynamic trace data were aggregated over time, to arrive at static, frill 
course period accounts of trace data. Second, from the large array of trace variables, a selection was made by 
focusing on process variables most strongly connected to alternative pedagogical behaviours of students. 
These include the alternative feedback modes preferred by students. In total, six trace variables were 
selected: 

• #Attempts : total number of attempts of individual exercises; 

• #Examples : total number of worked examples called; 

• #AttemptsCorrect : total number of attempts with correct answers; 

• #AttemptsInCorr: total number of attempts with incorrect answers; and 

• Mastery, the proportion of the in total 160 exercises successfully answered. 

In order to disentangle the effects of learning intensity from different learning approaches, we restricted 
the sample to those students who have been very active in the e-tutorial and achieved at least a 70% mastery 
level (that is, successfully solved at least 112 of the 160 exercises): 593 of the 1080 students. From this 
subset about 20% never used any worked example during the course. The other students were split into three 
equal-sized groups according to intensity of using worked examples. Table 1 provides descriptive statistics of 
these four sub-samples. #Exerc indicates the average number of exercises for which students called at least 
one example, and #Example/Exerc provides the average number of examples called in these exercises. 


Table 1. Descriptive statistics of four sub-samples of students achieving at least 70% mastery level 


Group 

N 

Mastery 

1 

1 

#Examples 

#Exerc 

#Example/Exerc 

NoExamples 

114 

90% 

990 

0 

0 

0 

SomeExamples 

159 

91% 

1211 

34 

12 

2.9 

AverageExampl 

159 

93% 

1335 

104 

35 

3.0 

IntensiveExampl 

161 

93% 

1719 

242 

57 

4.3 

Total 

593 

92% 

1340 

103 

28 

3.7 


Mastery> level is indeed invariant over groups. #Attempts, #Examples, #Exercises where examples are 
called, and #Example/Exerc do strongly increase over the four sub-samples. Since all observations are of 
count-type, they are all strongly right skewed. To diminish skewness and achieve better normality, we 
re-express all count data into square root transforms when applying correlation or regression analyses. As 
quantitative measures of how intensive students have used worked examples, we defined #ExercExamplSqrt 
as the square root of the number of exercises in which students called at least one example and 
#ExercMultExamplSqrt as the number of exercises in which the student called multiple examples, negatively 
exponentially weighted with the number of examples called. 

In this study, we will make a selection with regard to the self-report surveys measuring student learning 
dispositions. More than a dozen were administered, ranging from affective learning emotions to cognitive 
learning processing strategies. We will focus here on a selection of six instruments measuring aspects of 
self-regulated learning (SRL), expectancy-value based learning attitudes, national culture dimensions and 
learning emotions, since these dispositions have been investigated in recent LA studies (see Azevedo et ah, 
2013; Duffy & Azevedo, 2015, Mittelmeier, Tempelaar, Rienties, & Nguyen, 2016 and references therein). 
All disposition surveys are measured using seven-point Likert scales; no transformations of variables were 
required except a logarithmic transform for the Effort variable. 

In the characterisation of national cultural differences, research by Hofstede (Hofstede, Hofstede, & 
Minkov, 2010) takes a prominent position. Based on an analysis of attitude surveys obtained from employees 
in more than 50 countries, Hofstede identified six major dimensions on which cultures differ. Power distance 
(PDF) refers to the extent to which less powerful members of organisations and institutions accept and expect 
unequal distribution of power. Uncertainty avoidance ( UAP) refers to society’s tolerance for uncertainty and 
ambiguity. Individualism versus collectivism ( IND ) signals the degree to which individuals are integrated 
into groups: from loose ties between individuals and self-agency to integrated and strong, cohesive societies. 
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In masculine societies (MAS), emotional gender roles are rather distinct, whereas, in feminine societies, these 
roles overlap. Long-term orientation ( TOWVS ) distinguishes societies in being directed towards future 
rewards, or the fulfilment of present needs and desires. The final and most recently added cultural dimension 
is that of indulgence versus restraint (IVR) and signals the degree to which a culture allows or suppresses 
gratification of needs. In research applying national culture differences, students are assigned culture 
dimension scores based on their nationality (Mittelmeier, Tempelaar, Rienties, & Nguyen, 2016). 

Preferred processing strategies of students, part of a broader array of self-regulated learning dispositions 
(Vermunt, 1996), allow for an ordinal classification from two deep learning orientations, Critical processing, 
and Relating and structuring, through Concrete processing, to two surface or step-wise learning orientations: 
Analysing and Memorising. 

Expectancy-value based attitudes towards learning of statistics were assessed with the SATS instrument 
(Tempelaar, Gijselaers, Schim van der Loeff, & Nijhuis, 2007). The instrument contains six attitudes: Affect: 
students’ feelings concerning mathematics; C ognComp: students’ self-perceptions of their intellectual 
knowledge and skills when applied to mathematics; Value: students attitudes about the usefulness, relevance, 
and worth of mathematics in personal and professional life; NoDifficulty: students’ perceptions that 
mathematics as a subject is not difficult to learn; Interest: students’ level of individual interest in 
mathematics; and Effort: the amount of work the student is willing to undertake to learn mathematics. 

The Control-Value Theory of Achievement Emotions (CVTAE, Pekrun, 2006) postulates that emotions 
that arise in learning activities differ in valence, focus, and activation. Emotional valence can be positive 
(enjoyment) or negative (anxiety, hopelessness, boredom). CVTAE describes the emotions experienced in 
relation to an achievement activity (e.g. boredom experienced whilst preparing homework) or outcome (e.g. 
anxiety towards performing at an exam). The activation component describes emotions as activating (i.e. 
anxiety leading to action) versus deactivating (i.e. hopelessness leading to disengagement). From the 
Achievement Emotions Questionnaire (AEQ, Pekrun, Gotz, Frenzel, Barchfeld, & Perry, 2011) measuring 
learning emotions we selected four scales: positive activating emotion Enjoyment, negative activating 
emotion Anxiety, neutral deactivating Boredom and negative deactivating Hopelessness. Next, Academic 
Control is included as the antecedent of all learning emotions. Learning emotions of epistemic type are 
related to cognitive aspects of the task itself (Pekrun, 2012). Prototypical epistemic emotions are curiosity 
and contusion. In this study, epistemic emotions were measured with the Epistemic Emotion Scales (EES, 
Pekrun & Meier, 2011), including Surprise, Curiosity, Confusion, Anxiety, Frustration, Enjoyment, and 
Boredom. 

Course performance data is based on the final written exam, as well as the three intermediate quizzes. 
Quiz scores are averaged, and both exam and quiz are decomposed into two topic scores, resulting in 
MathExam, StatsExam, MathQuiz and StatsQuiz. 


4 . RESULTS 


Grouping based on intensity of example use induces statistically significant but small group differences for 
gender (sign=.03, eta squared=1.5%), math prior education (sign-009, eta squared=2.0%), and international 
students (sign<.001, eta squared=3.4%); see the left panel of Figure 2. 



Figure 2. Group differences for demographics (left) and culture dimensions (right) 
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The same conclusion is reached for national culture dimensions: statistically significant but small group 
differences exist for Power distance (sign=.007, eta squared=2.0%), Masculinity (sign<.001, eta 
squared=4.0%), Uncertainty avoidance (sign<.001, eta squared=4.6%), Long-term orientation (sign—.014. 
eta squared - 1.8%), and Indulgence versus restraint (sign - .001, eta squared=2.7%). Similar size differences 
are visible in the deep learning facets of the cognitive processing strategies: Critical processing (sign=.002, 
eta squared=2.6%), Relating and structuring (sign - 002, eta squared=2.6%), resulting in the aggregated 
group difference of the Deep learning scale being significant (sign<.001, eta squared=3.1%). Also, the 
Concrete processing scale demonstrates group differences (sign - .013, eta squared=1.8%), but not the two 
scales representing Surface learning approaches, Analysing, and Memorizing (Figure 3, left panel). 


5.2 -i- 7.0 



Figure 3. Group differences for cognitive processing strategies (left) and learning attitudes (right) 

Learning attitudes exhibit group differences for Affect (sign<.001, eta squared=6.5%), Cognitive 
competence (sign<.001, eta squared=5.1%), Value (sigiK.001, eta squared=3.8%), Interest (sign - .032, eta 
squared=1.5%) and Effort (sign<.024, eta squared - !.6%). but not for NoDifficulty (Figure 3, right panel). 
The effect of positive affect in the attitude scale is repeated in the two learning emotion scales. Epistemic 
emotions demonstrate group differences for the negative emotions Confusion (sign<.001, eta squared=3.3%), 
Anxiety (sigiK.001, eta squared - 4.8%), and Frustration (sign<.001, eta squared=3.6%), and the positive 
emotion Enjoyment (sign=,015, eta squared=1.8%): Figure 4, left panel. Emotions in the actual doing of 
learning activities, visible in the right panel of Figure 4, demonstrate even larger differences for Academic 
control (sign<.001, eta squared=5.0%), learning Anxiety (sign<.001, eta squared=5.9%), and learning 
Hopelessness (sign<.001, eta squared=5.2%), blit not for learning Boredom and Enjoyment. 
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Figure 4. Group differences for epistemic emotions (left) and learning achievement emotions (right) 

Last group differences refer to intensity of using the e-tutorial, and course performance: Figure 5. 
Significant differences exist in the square root transforms of #Attempts (sign<.001, eta squared=17.8%), 
#AttemptsCorrect (sign<.001, eta squared=3.0%), HAttemptsInCorr (sign<.001, eta squared=32.4%), as well 
as course performance measures MathQuiz (sign<.001, eta squared=7.3%), StatsQuiz (sign<.001, eta 
squared=14.0%), MathExam (sign<.001, eta squared=10.5%), and StatsExam (sign<.001, eta 
squared=13.0%). Outcomes of correlational analyses point in the same direction. The index for multiple use 
of examples, exponentially weighting for the number of repeated examples, correlates -.10 with Critical 
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processing, -.07 with Concrete processing, .08 with Relating, .10 with Memorizing, .11 and .13 with 
epistemic Anxiety and Frustration, .13 with Gender, and -.15 for StatsExam, but not with MathExam (all 
significance levels below .01). In a multivariate context, the best prediction equation for multiple usage of 
examples, by step-wise regression, in standardized coefficients, is: #ExercMultExamplSqrt = . 1 ^Masculinity 
+ .07 UncertaintyA\ + .09 Female - ,06CriticalPr -AOAnalysing + .12 Memorizing +.\2Ejfort - .09 Value with 
R 2 = .32. 

1.0 
0.9 
0.8 
0.7 
0.6 
0.5 
0.4 
0.3 
0.2 


Figure 5. Group differences for e-tutorial activity (left) and course performance (right) 



Attempts AttCorrect AttlnCorr MathQuiz StatsQuiz MathExam StatsExam 


5. DISCUSSION AND CONCLUSION 

The same mechanisms that explain why worked examples constitute an effective way of learning appear to 
explain individual differences in the preference for learning with worked examples when students can freely 
choose between alternative didactical scenarios. The danger of cognitive overload in the problem-solving 
scenario to strengthen a surface orientation of learning and to weaken the deep learning orientation of 
students (Renkl, 2014) is seen as a major merit of example-based learning. Exactly these same factors, the 
deep versus surface learning orientations, pop up as factors that explain individual differences in learning 
preferences. The Critical processing scale, most indicative for deep learning, is negatively related to the use 
of worked examples, whereas Memorizing, most indicative for surface learning, is positively related to the 
use of worked examples. But profiting from the availability of a large set of learning dispositions, we find 
that several other factors play a role, mostly of affective type. Adaptive antecedents as Affect and Value are 
negatively related to the use of examples, whereas maladaptive antecedents as epistemic Confusion, Anxiety 
and Frustration, and learning emotions LAnxiety and Hopelessness, are positively related to the use of 
worked examples. And in our international context, national culture impacts students learning preferences 
too, mainly through Masculinity and UncertaintyAvoidance being positively related to the use of examples. 
In total this suggests that where the use of (multiple) worked examples may help prevent students turning 
into maladaptive learning approaches because of cognitive overload, this help does not address all students in 
the same way. It is especially the group of students who score high on maladaptive dispositions, who are best 
helped by worked examples, whilst for a large category of students who opt out, using the examples only 
very infrequently or even not at all, worked examples seem to be of no added value. What suggests that a 
high level of adaptive learning dispositions protects these students from cognitive overload. 

The story is richer than learning dispositions only. We do find gender and national culture effects: female 
students, and students from masculine and uncertainty avoiding cultures, do use worked examples more 
frequently than other students. The cultural facet in this may signal issues of adaptation of international 
students (Mittelheimer et al., 2016). The Dutch society, its educational systems, and PBL in specific, are 
strongly characterised by femininity and low levels of uncertainty avoidance and power distance. 
International students from cultures with opposite characteristics may choose to intensively use worked 
examples in their adaptation to the student-centred, PBL type of instruction they are so unfamiliar with. 

The last conclusion relates the remarkable fact that intensity of usage of worked examples is as predictive 
for mathematics-related performance, as it is for statistics related performance. All worked examples in MSL 
focus on statistics topics; no one addresses any math topic. So from a cognitive perspective, there cannot be 
any effect from studying these worked examples on math performance. The only explanation that remains is 
that the frequent use of worked examples indeed signals higher levels of maladaptive dispositions. 
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This study is first and for all a showcase on the role of DLA in the provision of personalized, detailed 
learning feedback. No other program will apply the DLA with the same set of dispositions as described here. 
It is the general ability to identify students most in need of extra learning scaffolds that is the main lesson. 
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